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PURPOSE 


Pursuant to Consent Decree Paragraph 137, the Office of the Inspector General (OIG) conducted 
a Training Evaluation and Management System (TEAMS) II audit. Specifically, Paragraph 137 
requires the Inspector General to audit the quality and timeliness of the Los Angeles Police 
Department’s (LAPD or Department) use of TEAMS II and to evaluate the Department’s 
compliance with Consent Decree Paragraph 47, which is the protocol for using TEAMS II. 
Paragraph 47 contains 13 sub-paragraphs [(a) to (m)], the following four (a, b, i and m) which 
were the focus of this audit: 


Paragraph 47(a) — “The protocol shall require that, on a regular basis, supervisors review and 
analyze all relevant information in TEAMS II about officers under their supervision to detect any 
pattern or series of incidents that indicate that an officer, group of officers, or an LAPD unit 
under his or her supervision may be engaging in at-risk behavior. ” 





Paragraph 47(b) — “The protocol shall provide that when at-risk behavior may be occurring 
based on a review and analysis described in the preceding subparagraph, appropriate managers 
and supervisors shall undertake a more intensive review of the officer's performance.” 





Paragraph 47(i) — “The protocol shall provide that managers’ and supervisors' performance in 
implementing the provisions of the TEAMS II protocol shall be taken into account in their annual 
personnel performance evaluations.” 


Paragraph 47(m) — “The protocol shall require that whenever an officer transfers into a new 
Division or Area, the Commanding officer of such new Division or Area shall promptly cause the 
transferred officer's TEAMS II record to be reviewed by the transferred officer's watch 
commander or supervisor.” 


Note: One important provision of the TEAMS II protocol is that managers and supervisors 
perform the required timely reviews of subordinate officers’ TEAMS II reports and work 
history to identify subordinate officers with potential patterns of at-risk practices needing 
corrective action. Conversely, these reviews should also identify subordinate officers 
deserving commendations. 


BACKGROUND 


This is the fourth TEAMS II audit conducted by the OIG. The first and third TEAMS II audits 
(both Phase I) primarily focused on threshold-activated Action Items.' A threshold-activated 
Action Item is automatically generated in TEAMS II when an employee has an inordinate 
number of uses of force, complaints, vehicle pursuits, traffic collisions, or claims/lawsuits in 
comparison to his/her peer group average. Indeed, the Department’s implementation of the 
threshold-activated Action Items process is the “heart” of the early warning system required by 
the Consent Decree. It allows Department supervisors and managers to review an employee’s 





' The related audit reports were issued on November 6, 2007 and October 2, 2008, respectively. 
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activity to evaluate whether the employee is exhibiting any type of potential at-risk behavior that 
should be addressed or monitored. 

The second TEAMS II audit and this audit (both Phase II) primarily focused on Action Items 
generated for sworn officer transfers and annual performance evaluations, and Action Items 
created at the discretion of supervisors and managers for follow-up and monitoring purposes. 
The prior TEAMS II Phase II audit concluded that Action Items were generally not being 
manually created, as required, for sworn officer transfers and annual performance evaluations. 
To remedy this problem, TEAMS II Development Bureau implemented major procedural and 
system changes, whereby Transfer Action Items (TAIs) were to be automatically generated 
beginning April 16, 2008 for all sworn officer transfers, and Performance Evaluation Report 
Action Items (PERs) were to be automatically generated beginning May 1, 2008 for all sworn 
officer annual performance evaluations. Hence, one major objective of this audit was to 
determine if TAIs and PERs were indeed being automatically generated, as required. 


SUMMARY OF AUDIT SCOPE AND METHODOLOGY 


The details of the scope and methodology for each audit objective are provided in the Detailed 
Results of the Audit section of this report. However, the bulk of the audit testing was performed 
on samples of Action Items closed in November 2008, the most current month available during 
the planning phase of the audit. These Action Item samples included 26 Transfer Action Items 
(TAIs), 22 Performance Evaluation Report Action Items (PERs), and 24 Supervisor Action Items 
(SAIs). 


All TAIs completed from September 1 to November 30 of 2008 were analyzed to determine the 
number of days it took to complete them. All TAIs and PERs that were still open at December 3, 
2008, were analyzed to determine the number of days they remained open. 


Note: All audit sample sizes, unless otherwise indicated in the Detailed Results of the Audit 
section of this report, were calculated based on a 95 percent one-tail confidence level, an 
expected error rate of six percent, and a plus-precision of seven percent. 


AUDIT OBJECTIVES 
A. Transfers 


Objective Al: To determine if Transfer Action Items (TAIs) were being generated for all sworn 
officer transfers. 


Objective A2: To determine if both the TAIs and TEAMS Evaluation Reports (TERs} were 
completed within 10 days of publication of the related transfer orders. 





? The TER, a manual form, is required to be completed for each transfer and loan. The TER shall document that the 
reviewer has identified and analyzed the employee’s applicable sustained complaints, adverse judicial findings, and 
uses of force. The TER shall be approved by both the watch commander and commanding officer who receive the 
transferred officer. 
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Objective A3: To determine if management approved the TERs, adequately discussed any 
sustained complaints and uses of force, and used current TEAMS II reports in the evaluations. 


B. Annual Performance Evaluations 


Objective B1: To determine if Performance Evaluation Report Action Items (PERs) were being 
generated for all sworn officer annual performance evaluations. 


Objective B2: To determine if both the PERs and hardcopy Performance Evaluation Reports 
were completed within 60 days of the annual anniversary date of the employee’s current rank, 
regardless of paygrade.* 


Objective B3: To determine if either the PERs or the hardcopy Performance Evaluation Reports 
documented that the manager’s or supervisor’s performance in implementing the provisions of 
the TEAMS II protocol was taken into account in their annual performance evaluation. 


C. Supervisor Action Items (SAIs) 


Objective C: To determine if the disposition assigned for each SAI was consistent with 
information in the sworn officer’s TEAMS II report and if the applicable corrective action was 
taken. 


SUMMARY OF RESULTS FOR EACH AUDIT OBJECTIVE 


Objective Al: TAIs were generated for all 97 (100%) of the sworn officer transfers tested. 


Objective A2: Of the total 653 TAIs completed from September 1 to November 30 of 2008, 623 
(95%) were completed either on time or were less than 31 days late. Also, 180 TAIs were open 

(pending completion) as of December 3, 2008; and 26 of these 180 (14%) open TAIs were more 
than 60 days past due. 


Objective A3: For all 26 (100%) TAIs in the audit sample tested, the supervisor and 
commanding officer approved the TER; adequately addressed employee’s applicable sustained 
complaints, adverse judicial findings, and uses of force; and used a timely TEAMS II report in 
the evaluation. 


Objective B1: PERs were generated for all 30 (100%) sworn officers in the audit sample with 
November anniversary dates. 


Objective B2: Of the 22 hardcopy Performance Evaluation Reports in the audit sample tested, 
10 (45%) were completed on time, 11 (50%) were completed late (averaging 27 days late), and 1 





? For example, if the employee was promoted to his/her current rank of Sergeant on September 16, 2006, the current 
rank anniversary date for 2008 would be September 16, 2008. 

* For example, if the No Action disposition was assigned, there were no recent sustained complaints, out-of-policy 
uses of force, preventable traffic collisions, etc. that were not already addressed which might have warranted a 
different disposition. 
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(5%) was not completed. Also, 2,101 PERs were open (pending completion) as of December 3, 
2008; and 11 (0.5%) of these 2,101 open PERs were more than 60 days past due. 


Objective B3: For 11 of the 18 (61%) applicable PERs in the audit sample tested, there was 
evidence that the manager’s or supervisor’s performance in implementing the provisions of the 
TEAMS II protocol was taken into account in their annual performance evaluation. 


Note: Effective January 2009 (subsequent to audit testing), the Department replaced the 
hardcopy Performance Evaluation Report form with the Standards Based Assessment (SBA) 
form, which prompts the evaluator to address the supervisor’s or manager’s performance in 
implementing the provisions of the TEAMS II protocol. Hence, the compliance rate should 
greatly improve in the future. 


Objective C: The disposition assigned for each of the 24 SAIs in the audit sample tested was 
consistent with information in the sworn officer’s TEAMS II report. 


The Commanding Officer of TEAMS II Development Bureau provided a written response to a 
draft of this Report and expressed agreement with the Audit’s results. The OIG wishes to thank 
TEAMS II Development Bureau for their cooperation and sharing of their expertise while we 
were conducting this Audit. 


DETAILED RESULTS OF THE AUDIT 


A. TRANSFERS 


Background and References 


Consent Decree Paragraph 47(m) — “The protocol shall require that whenever an officer 


transfers into a new Division or Area, the Commanding officer of such new Division or Area 
shall promptly cause the transferred officer's TEAMS II record to be reviewed by the transferred 
officer's watch commander or supervisor. ” 


Special Order No. 23, 2003, “Criteria for Transfers/Loans of Sworn Personnel Established” — 
This Special Order requires that the commanding officer gaining the transferred officer ensure 


that the watch commander or officer in charge review the sworn officer’s TEAMS II report 
within ten days of publication of the related transfer order and document this review on the 
sworn officer’s TEAMS Evaluation Report (TER). The TER shall document that the reviewer 
has identified and analyzed the employee’s applicable sustained complaints, adverse judicial 
findings, and uses of force. 


Intradepartmental Correspondence (15.2) dated April 18, 2008 from the Commanding Officer of 
TEAMS II Development Bureau to all commanding officers — This 15.2 advises that TEAMS II 


will electronically generate a Transfer Action Item (TAI) for each sworn officer transferring into 
acommand. The officer’s supervisor shall complete the TAI by reviewing the officer’s TEAMS 
II report and referencing the TER, and the officer’s commanding officer shall approve and 
complete the TAI and TER within 10 days of publication of the related transfer order. 
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Audit Objectives 


Objective Al: To determine if Transfer Action Items (TAIs) were being generated for all sworn 
officer transfers. 


Objective A2: To determine if both the TAIs and TEAMS Evaluation Reports (TERs)° were 
completed within 10 days of publication of the related transfer orders. 


Objective A3: To determine if management approved the TERs, adequately discussed any 
sustained complaints and uses of force, and used current TEAMS II reports in the evaluations. 


Scope and Methodology 


Objective Al: For each of the 39 sworn officer promotions on Transfer Order Nos. 12 and 13 of 
2008, it was determined if a corresponding TAI was generated. For each of the 58 judgmentally- 
selected sworn officer transfers and inter-divisional paygrade changes on Transfer Order Nos. 12 
and 13 of 2008, it was determined if a corresponding TAI was generated. ° 


Objectives A2 and A3: From the population of all 135 TAIs closed in November 2008, 26 TAIs 
were randomly selected for the audit sample. Testing included examination of the corresponding 
Transfer Orders, TAIs, TERs and TEAMS II reports. 


Results 
Objective Al: TAIs were generated for all 97 (100%) of the sworn officer transfers tested. 


Objective A2 - Timeliness of Completion Tests: Twenty (20) of the 26 (77%) TAIs tested were 
completed within 10 days of publication of the related transfer order. The remaining 6 TAIs 
were late anywhere from 2 to 33 days. The OIG expanded the scope to examine all 653 TAIs 
completed from September 1 to November 30 of 2008, as listed on a report received from 
TEAMS II Development Bureau. The OIG noted that 464 of the 653 TAIs (71%) were 
completed on time and that 623 (95%) were completed either on time or less than 31 days late: 
































>90 days late 61-90 days late 31-60 days late 1-30 days late_| on time total 
3 7 20 159 464 653 
0.4% 1.1% 3.1% 24.3% 71.1% 100% 











` The TER, a manual form, is required to be completed for each transfer and loan. The TER shall document that the 
reviewer has identified and analyzed the employee’s applicable sustained complaints, adverse judicial findings, and 
uses of force. The TER shall be approved by both the watch commander and commanding officer who receive the 
transferred officer. 

é The sample of 58 consists of all sworn supervisory inter-divisional paygrade changes and transfers listed on 
Transfer Order Nos. 12 and 13 of 2008, and every tenth sworn non-supervisory inter-divisional paygrade change or 
transfer listed on Transfer Order No. 12 of 2008. Note that all the transfers, promotions and inter-divisional 
paygrade changes on the Transfer Orders involve officers transferring from one division to another, so TAIs are 
required. 
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Twenty-three (23) of the 26 (88%) TERs tested were completed within 10 days of publication of 
the related Transfer Order. The remaining 3 TERs were late anywhere from 2 to 31 days. 

For 19 of the 26 of the transfers tested, there was a difference between the TAI and TER 
completion dates, ranging from: 

e The TER completion date succeeding the TAI completion date by 34 days, to 

e The TAI completion date succeeding the TER completion date by 37 days. 

The net average result for all 26 transfers was that the TAI completion date succeeded the TER 
completion date by 2.7 days. 


Objective A2 — Aging of Open (Pending Completion) TAIs: One hundred and eighty (180) TAIs 
were open (pending completion) as of December 3, 2008; and 26 of these 180 (15%) open TAIs 


were more than 60 days past due. 

















>90 days | 61-90 days | 31-60 days | 1-30 days | not past 
past due past due past due past due due total 
21 5 21 24 109 180 
12% 3% 12% 13% 61% 100% 




















Objective A3 — Testing for Proper Approval and Completion of TERs: For all 26 (100%) TAIs 
in the audit sample tested, the supervisor and commanding officer approved the TER; adequately 


addressed employee’s applicable sustained complaints, adverse judicial findings, and uses of 
force; and used a timely TEAMS II report in the evaluation. 


B. ANNUAL PERFORMANCE EVALUATIONS 


Background and References 


Consent Decree Paragraph 47(1) — “The protocol shall provide that managers’ and supervisors’ 
performance in implementing the provisions of the TEAMS II protocol shall be taken into 
account in their annual personnel performance evaluations.” 


Intradepartmental Correspondence (15.2) dated April 18, 2008 from the Commanding Officer, of 
TEAMS II Development Bureau to all commanding officers — This 15.2 advises that TEAMS II 


will electronically generate a Performance Evaluation Report Action Item (PER) for each sworn 
officer 30 days before their annual anniversary of their current rank. This PER and the related 
hardcopy Performance Evaluation Reports shall be completed within 60 days of the officer’s 
anniversary date. 





Audit Objectives 


Objective B1: To determine if PERs were being generated for all sworn officer annual 
performance evaluations due. 
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Objective B2: To determine if both the PERs and hardcopy Performance Evaluation Reports 
were completed within 60 days of the annual anniversary date of the employee’s current rank, 
regardless of paygrade.’ 


Objective B3: To determine if either the PERs or the hardcopy Performance Evaluation Reports 
documented that the manager’s or supervisor’s performance in implementing the provisions of 
the TEAMS II protocol was taken into account in their annual performance evaluation. 


Scope and Methodology 


Objective B1: From the population of all 666 sworn officers with November anniversary dates, 
30 officers were randomly selected for the audit sample. For each officer, it was determined if a 
corresponding PER was generated. 


Objectives B2 and B3: From the population of all 80 PERs for sworn supervisory and 
managerial officers closed in November 2008, 22 PERs were randomly selected for the audit 
sample. Testing included examination of the PERs and hardcopy Performance Evaluation 
Reports. 





Results 


Objective B1: PERs were generated for all 30 (100%) sworn officers in the audit sample with 
November anniversary dates. 


Objective B2 - Timeliness of Completion Tests: Of the 22 hardcopy Performance Evaluation 
Reports in the audit sample tested, 10 (45%) were completed on time, 11 (50%) were completed 
late (averaging 27 days late), and 1 (5%) was not completed. Also, 2,101 PERs were open 
(pending completion) as of December 3, 2008; and 11 (0.5%) of these 2,101 open PERs were 
more than 60 days past due. 


According to TEAMS II Development Bureau, timeliness of completion should be determined 
based on the date that the evaluated officer signs the hardcopy Performance Evaluation Reports 
(hardcopy date), not the completion date recorded on the related PER (electronic date). While 
these dates may differ, what is important is that the employee is served timely. 

e For 13 PERs, the hardcopy date succeeded the electronic date, ranging from 1 to 105 days. 
e For 7 PERs, the electronic date succeeded the hardcopy date, ranging from 4 to 131 days. 

e For 1 PER, the hardcopy and electronic dates were the same. 

The net average result for the 21 completed PERs was that the hardcopy date succeeded the 
electronic date by 10 days. 





’ For example, if the employee was promoted to his/her current rank of sergeant on September 16, 2006, the current 
rank anniversary date for 2008 would be September 16, 2008. 
8 The random sample of 22 included 21 supervisors (19 sergeants and 2 lieutenants) and 1 manager (captain). 
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Objective B2 — Aging of Open (Pending Completion) PERs: Of the 2,101 PERs open as of 
December 3, 2008, 392 (19%) were past due. Note that only 11 of the 2,101 (0.5%) open PERs 


were more than 60 days past due. See schedule below. 





























>90 days | 61-90 days | 31-60 days | 1-30 days | not past 
past due past due past due past due due Total 
0 11 132 249 1,709 2,101 
0% 0.5% 6.3% 11.9% 81.3% 100% 











Objective B3 — Testing for Compliance with 47(1): For 11 of the 18 (61%) applicable PERs 
tested in the audit sample tested,’ there was evidence that the manager’s or supervisor’s 
performance in implementing the provisions of the TEAMS II protocol was taken into account in 
their annual performance evaluation.'” 


Note: Effective January 2009, the Department replaced the hardcopy Performance 
Evaluation Report form with the Standards Based Assessment (SBA) form. The SBA form’s 
Development of Subordinates section contains the following boxes to be checked if the 
officer either meets or sometimes exceeds standards (10)," or needs improvement (2(1): 
10 “Regularly reviews various RMIS” reports, audits, Action Items using the TEAMS 
system. Completes TEAMS II transfer reviews on time.” 

20 “Seldom or never reviews various RMIS reports, audits, Action Items using the TEAMS 
II system. Does not complete the TEAMS II transfer reviews on time.” 





















































Hence, presuming that the reviewers check boxes 10 or 2L as required, compliance with 
Consent Decree Paragraph 47(i) should improve in the future. 





C. SUPERVISOR ACTION ITEMS (SAIs) 


Background and References 


Consent Decree Paragraph 47(a) — “The protocol shall require that, on a regular basis, 
supervisors review and analyze all relevant information in TEAMS II about officers under their 
supervision to detect any pattern or series of incidents that indicate that an officer, group of 
officers, or an LAPD unit under his or her supervision may be engaging in at-risk behavior. ” 





Consent Decree Paragraph 47(b) — “The protocol shall provide that when at-risk behavior may 
be occurring based on a review and analysis described in the preceding subparagraph, 
appropriate managers and supervisors shall undertake a more intensive review of the officer's 
performance.” 





? Some of the officers included in the sample with a supervisory rank (e.g., an Internal Affairs Group sergeant 
investigating complaints) did not actually supervise officers. Hence, most of the provisions of the TEAMS II 
protocol, including reviewing Action Items, were not applicable. 

10 The Federal Monitor reported 53% compliance with CD] 47(i) in its quarterly report issued November 17, 2008, 
pertaining to its audit testing of PERs completed in Deployment Periods Nos. 7 and 8 of 2008. 

' If the officer greatly exceeds standards, box 1 should be checked, with a supplemental narrative to explain why, 
as there is no comparable box in the greatly exceeds standards performance level. 

'? Risk Management Information System (RMIS) is an application within TEAMS II. 














Training Evaluation and Management System (TEAMS) II Audit, Phase II 
Page 10 of 10 
1.0 


Special Order No. 28, 2008, dated August 29, 2008 - This Special Order states that TEAMS II 
allows supervisors and managers to manually create a SAI to review a subordinate’s 
performance and to document reviews for periodic monitoring. The supervisor should document 
the extent of their review and justify any corrective action necessary. Also, an SAI can be 
created if the result of a subordinate’s annual performance evaluation warrants further action. 


Audit Objective C 


To determine if the disposition assigned for each SAI was consistent with information in the 
sworn officer’s TEAMS II report and if the applicable corrective action was taken. '° 


Scope and Methodology 


From the population of all 94 SAIs closed in November 2008,'* 24 SAIs were randomly selected 
for the audit sample. For each SAI, the officer’s TEAMS II and Summary of Employee Activity 
reports’? were generated and examined. 


Results 


The disposition assigned for each of the 24 SAIs in the audit sample tested was consistent with 
information in the sworn officer’s TEAMS II report. Testing for evidence of corrective action 
was not applicable for any of the 24 SAIs, as none of the 24 SAIs had dispositions indicating that 
corrective action was needed. 


CONCLUSION 


TAIs were generated as required for all sworn officer transfers tested, and PERs were generated 
as required for all sworn officers with November 2008 rank anniversary dates tested. This was a 
major improvement since the last audit, which found that action items were not created 
[manually] for the vast majority of sworn officer transfers and annual performance evaluations. 


The TEAMS Evaluation Reports (TERs) pertaining to the TAIs tested were completed accurately 
and reasonably timely. The hardcopy Performance Evaluation Reports pertaining to the PERs 
were also generally completed reasonably timely. 





13 For example, if the No Action disposition was assigned, there were no recent sustained complaints, out-of-policy 
uses of force, preventable traffic collisions, etc. that were not already addressed which might have warranted a 
different disposition. 

14 The population of 94 SAIs excluded those SAIs with the dispositions of Cancel and No Action — Note for Annual 
Review Only. 

15 The Summary of Employee Activity Report (R02) compares the employee’s activity numbers (of arrests, 

citations, claims and lawsuits, collisions, complaints, etc.) to the averages within their unit and two other comparable 
units. 
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Sixty-one percent of the sworn supervisor or manager annual evaluations tested included 
evidence on either the PER or hardcopy Performance Evaluation Reports that the supervisor’s or 
manager’s performance in implementing the provisions of the TEAMS II protocol was taken into 
account in their annual personnel performance evaluation, as required by Consent Decree 
Paragraph 47(i). Effective January 2009 (subsequent to audit testing), the Department replaced 
the hardcopy Performance Evaluation Reports with the Standards Based Assessment (SBA) 
form. Since this new SBA form prompts the evaluator to address the supervisor’s or manager’s 
performance in implementing the provisions of the TEAMS II protocol, compliance with 
Consent Decree Paragraph 47(i) should improve in the future. 


